Deep Apprenticeship Learning for Playing Video Games

Recently it has been shown that deep neural networks can learn to play Atari games by directly observing raw pixels of the playing area. We show how apprenticeship learning can be applied in this setting so that an agent can learn to perform a task (i.e. play a game) by observing the expert, without any explicitly provided knowledge of the game’s internal state or objectives. Background Mnih et al. (2013) recently demonstrated that it is possible to combine Q-learning with deep learning to play Atari games. Their method learns to maximize the score of the game, which is explicitly provided to the model during training. We extend the approach of Mnih et al. (2013) to the apprenticeship learning setting, allowing our agent to learn to play without being provided with any explicit knowledge of the game score. In this paper we take a very simple approach to apprenticeship learning by simply observing the expert play and training a classifier to identify expert actions from game states. Deep Apprenticeship Learning Following Mnih et al. (2013), we train a convolutional neural network to play Atari games by observing only raw pixels of the playing area. However, instead of learning to maximize the game score directly, we attempt to imitate the behaviour of an expert player. By watching an expert play, our network is able to learn to map game states to actions in a way that does not require that the score of the game is provided externally. We call our method Deep Apprenticeship Learning (DAL). Data Collection To interact with Atari games we used the Arcade Learning Environment (Naddaf 2010). To collect training data we modified the Arcade Learning Environment to record states (video frames) and actions (button presses) while a human plays the game. We collected human gameplay data for the Freeway game. Freeway is a classical game about trying to cross a street Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. actions (a) fully connected MLP convolutional layers state (s) Q(s, a1) Q(s, a2) Q(s, a3) Figure 1: Architecture of our deep apprenticeship learning network. while avoiding cars. Freeway is interesting from an RL perspective, because the agent only obtains a reward after crossing the street. That is, the reward occurs only after many actions are taken, and provided that the right actions are taken. This rare reward situation is very hard for RL systems. However, supervised learning, when expert data is available, can overcome this problem as shown in this paper. We collected 500,000 examples for Freeway. An example consists of a (state, action) pair, where for describing the state we used one, two or four sequential frames. However, we found that this choice did not have a significant effect on performance, which is sensible because the state of a game of Freeway can be entirely inferred from a single frame. We preprocess each of the game frames by converting the images from the 128 color Atari palette to grayscale. Each frame is resized from the original 210 × 160 to 83 × 83. Our preprocessing introduces some distortion of the original image, but based on visual inspection we believe that no relevant information is lost. We also remove the background from each frame by subtracting the median image computed over a large number of sample frames. (Figure 2)